Automated classification apparatus for shoulder disease via three dimensional deep learning method, method of providing information for classification of shoulder disease and non-transitory computer readable storage medium operating the method of providing information for classification of shoulder disease

ABSTRACT

An automated classification apparatus includes a 3D (three dimensional) Inception-Resnet block structure, a global average pooling structure and a fully connected layer. The 3D Inception-Resnet block structure includes a 3D Inception-Resnet structure configured to receive 3D medical image of a patient&#39;s shoulder and extract features from the 3D medical image and 3D Inception-Downsampling structure configured to downsample information of a feature map including the features. The global average pooling structure is configured to operate an average pooling for an output of the 3D Inception-Resnet block structure. The fully connected layer is disposed after the 3D global average pooling structure. The automated classification apparatus is configured to automatically classify the 3D medical image into a plurality of categories.

PRIORITY STATEMENT

This application claims priority under 35 U.S.C. § 119 to Korean PatentApplication No. 10-2019-0083387, filed on Jul. 10, 2019 in the KoreanIntellectual Property Office (KIPO), the contents of which are hereinincorporated by reference in their entireties.

BACKGROUND 1. Technical Field

Example embodiments relate to an automated classification apparatus forshoulder disease. More particularly, example embodiments relate to anautomated classification apparatus for shoulder disease via a threedimensional (3D) deep learning method.

2. Description of the Related Art

Diseases of a shoulder area may be diagnosed by a visual analysis of athree dimensional (3D) medical image such as magnetic resonance imagingor computed tomography imaging by a skilled specialist. It takes a lotof time, effort and experience to effectively analyze the 3D medicalimage. It is difficult to see a 3D image at a glance in the analysisprocess, so that the diagnosis may be concluded after repeatedlyobserving and analyzing multiple 2D images.

In conclusion, in the conventional shoulder disease diagnosis, it maytake a lot of time for diagnosis to secure high accuracy, and the resultof the diagnosis may depend on a personal skill of the specialistanalyzing an image.

SUMMARY

Example embodiments provide an automated classification apparatus for ashoulder disease capable of automatically classifying a degree of theshoulder disease via a three dimensional deep learning method.

Example embodiments provide a method of providing information ofclassification of the shoulder disease using the automatedclassification apparatus for the shoulder disease.

Example embodiments provide a non-transitory computer-readable storagemedium having stored thereon program instructions of the method ofproviding information of classification of the shoulder disease.

In an example automated classification apparatus for a shoulder diseaseaccording to the present inventive concept, the automated classificationapparatus includes a 3D (three dimensional) Inception-Resnet blockstructure, a global average pooling structure and a fully connectedlayer. The 3D Inception-Resnet block structure includes a 3DInception-Resnet structure configured to receive 3D medical image of apatient's shoulder and extract features from the 3D medical image and 3DInception-Downsampling structure configured to downsample information ofa feature map including the features. The global average poolingstructure is configured to operate an average pooling for an output ofthe 3D Inception-Resnet block structure. The fully connected layer isdisposed after the 3D global average pooling structure. The automatedclassification apparatus is configured to automatically classify the 3Dmedical image into a plurality of categories.

In an example embodiment, the plurality of the categories may include‘None’ which means that patient's rotator cuff tear is not present;‘Partial’, ‘Small’, ‘Medium’ and ‘Large’ according to a size of thepatient's rotator cuff tear.

In an example embodiment, the 3D medical image may sequentially passthrough a first 3D convolution structure, a first 3D Inception-Resnetblock structure, a second 3D Inception-Resnet block structure, a second3D convolution structure, the global average pooling structure and thefully connected layer.

In an example embodiment, the 3D Inception-Resnet block structure mayinclude three of the 3D Inception-Resnet structures and one of the 3DInception-Downsampling structure.

In an example embodiment, the 3D Inception-Resnet structure may includea first 3D convolution structure, a second 3D convolution structure anda third 3D convolution structure which are connected in series andforming a first path, a fourth 3D convolution structure and a fifth 3Dconvolution structure which are connected in series and forming a secondpath, a first concatenate structure configured to concatenate an outputof the third 3D convolution structure and an output of the fifth 3Dconvolution structure and an add structure configured to operate anelement-wise add operation of an output of the first concatenatestructure and an input of the 3D Inception-Resnet structure.

In an example embodiment, the 3D Inception-Downsampling structure mayinclude a sixth 3D convolution structure and a maximum pooling structureforming a third path, the maximum pooling structure configured to selecta maximum value from the output of the sixth 3D convolution structure, aseventh 3D convolution structure and an average pooling structureforming a fourth path, the average pooling structure configured toselect an average value from the output of the seventh 3D convolutionstructure, a first stride 3D convolution structure including aconvolution filter having an increased moving unit and forming a fifthpath, a second stride 3D convolution structure different from the firststride 3D convolution structure, including a convolution filter havingan increased moving unit and forming a sixth path and a secondconcatenate structure configured to concatenate an output of the maximumpooling structure, an output of the average pooling structure, an outputof the first stride 3D convolution structure and an output of the secondstride 3D convolution structure.

In an example embodiment, the automated classification apparatus mayfurther include a region of interest visualization part configured togenerate a heat map which visualizes a region of interest identified inthe 3D medical image in artificial intelligence generating a diagnosticresult of the 3D medical image.

In an example embodiment, the automated classification apparatus mayfurther include a 3D convolution structure disposed between the 3DInception-Resnet block structure and the global pooling averagestructure. The region of interest visualization part may be configuredto generate the heat map by multiplying first features which are outputof the 3D convolution structure and weights learned at the fullyconnected layer and summing multiplications of the first features andthe weights.

In an example embodiment, the heat map may be a 3D class activation map.

In an example method of providing information of classification ofshoulder disease according to the present inventive concept, the methodincludes receiving a 3D (three dimensional) medical image of a patient'sshoulder and extracting features from the 3D medical image, using a 3DInception-Resnet structure, downsampling information of a feature mapincluding the features, using a 3D Inception-Resnet block structure,operating an average pooling for an output of the 3D Inception-Resnetblock structure, using a global average pooling structure andautomatically classifying the 3D medical image into a plurality ofcategories.

In an example non-transitory computer-readable storage medium havingstored thereon program instructions, the program instructions executableby at least one hardware processor to receive a 3D (three dimensional)medical image of a patient's shoulder and extract features from the 3Dmedical image, using a 3D Inception-Resnet structure, downsampleinformation of a feature map including the features, using a 3DInception-Resnet block structure, operate an average pooling for anoutput of the 3D Inception-Resnet block structure, using a globalaverage pooling structure and automatically classify the 3D medicalimage into a plurality of categories.

According to the automated classification apparatus for the shoulderdisease may receive a 3D medical image and may analyze high dimensionalimages which a human cannot easily see at a glance using a 3D artificialintelligence algorithm based on a 3D CNN (convolutional neural network).The 3D artificial intelligence algorithm may learn by itself using alarge amount of images and big data regarding diagnostic recordsacquired previously. The 3D artificial intelligence algorithm mayrepresent diagnostic accuracy beyond a skilled orthopedist in a shortperiod.

In addition, the automated classification apparatus for the shoulderdisease of the present inventive concept may show a region of interestin medical images as a heat map in addition to accurately diagnosing theshoulder disease. The automated classification apparatus for theshoulder disease of the present inventive concept may generate a 3Dclass activation map to display regions of interest of the artificialintelligence and provide the 3D class activation map which is renderedin three dimensions as a supplementary information about a diagnosisresult.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other features and advantages of the present inventiveconcept will become more apparent by describing in detailed exampleembodiments thereof with reference to the accompanying drawings, inwhich:

FIG. 1 is a conceptual diagram illustrating a conventional diagnosisapparatus for a shoulder disease and an automated classificationapparatus for the shoulder disease according to an example embodiment ofthe present inventive concept;

FIG. 2 is a block diagram illustrating a three dimensional (3D)Inception-Downsampling structure according to an example embodiment ofthe present inventive concept;

FIG. 3 is a block diagram illustrating a 3D Inception-Resnet structureaccording to an example embodiment of the present inventive concept;

FIG. 4 is a block diagram illustrating a 3D Inception-Resnet blockstructure according to an example embodiment of the present inventiveconcept;

FIG. 5 is a block diagram illustrating the automated classificationapparatus for the shoulder disease according to an example embodiment ofthe present inventive concept;

FIG. 6 is a diagram illustrating an operation of a region of interestvisualization part of the automated classification apparatus for theshoulder disease according to an example embodiment of the presentinventive concept;

FIG. 7 is a screen shot illustrating an operation of the automatedclassification apparatus for the shoulder disease according to anexample embodiment of the present inventive concept;

FIG. 8 is a table illustrating MRI data of rotator cuff used in anexample embodiment of the present inventive concept;

FIG. 9 is a table illustrating diagnosis result of the automatedclassification apparatus for the shoulder disease according to anexample embodiment of the present inventive concept, diagnosis result oforthopedists specialized in shoulder and diagnosis result of generalorthopedists;

FIGS. 10 and 11 are graphs illustrating diagnosis result of theautomated classification apparatus for the shoulder disease according toan example embodiment of the present inventive concept, diagnosis resultof orthopedists specialized in shoulder and diagnosis result of generalorthopedists;

FIG. 12 is a diagram illustrating how the region of interest changes asartificial intelligence learning progresses in the automatedclassification apparatus for the shoulder disease according to anexample embodiment of the present inventive concept; and

FIG. 13 is a diagram illustrating rotator cuff tear data and region ofinterest visualization of the automated classification apparatus for theshoulder disease according to an example embodiment of the presentinventive concept.

DETAILED DESCRIPTION OF THE EMBODIMENTS

The present inventive concept now will be described more fullyhereinafter with reference to the accompanying drawings, in whichexemplary embodiments of the present invention are shown. The presentinventive concept may, however, be embodied in many different forms andshould not be construed as limited to the exemplary embodiments setfourth herein.

Rather, these exemplary embodiments are provided so that this disclosurewill be thorough and complete, and will fully convey the scope of thepresent invention to those skilled in the art. Like reference numeralsrefer to like elements throughout.

It will be understood that, although the terms first, second, third,etc. may be used herein to describe various elements, components,regions, layers and/or sections, these elements, components, regions,layers and/or sections should not be limited by these terms. These termsare only used to distinguish one element, component, region, layer orsection from another region, layer or section. Thus, a first element,component, region, layer or section discussed below could be termed asecond element, component, region, layer or section without departingfrom the teachings of the present invention.

The terminology used herein is for the purpose of describing particularexemplary embodiments only and is not intended to be limiting of thepresent invention. As used herein, the singular forms “a,” “an” and“the” are intended to include the plural forms as well, unless thecontext clearly indicates otherwise. It will be further understood thatthe terms “comprises” and/or “comprising,” when used in thisspecification, specify the presence of stated features, integers, steps,operations, elements, and/or components, but do not preclude thepresence or addition of one or more other features, integers, steps,operations, elements, components, and/or groups thereof.

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which this invention belongs. It will befurther understood that terms, such as those defined in commonly useddictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the relevant art andwill not be interpreted in an idealized or overly formal sense unlessexpressly so defined herein.

All methods described herein can be performed in a suitable order unlessotherwise indicated herein or otherwise clearly contradicted by context.The use of any and all examples, or exemplary language (e.g., “suchas”), is intended merely to better illustrate the invention and does notpose a limitation on the scope of the invention unless otherwiseclaimed. No language in the specification should be construed asindicating any non-claimed element as essential to the practice of theinventive concept as used herein.

Hereinafter, the present inventive concept will be explained in detailwith reference to the accompanying drawings.

FIG. 1 is a conceptual diagram illustrating a conventional diagnosisapparatus for a shoulder disease and an automated classificationapparatus for the shoulder disease according to an example embodiment ofthe present inventive concept.

Referring to FIG. 1, the conventional diagnosis apparatus for theshoulder disease divides a three dimensional (3D) medical image into aplurality of two dimensional (2D) images for the diagnosis when the 3Dmedical image is inputted.

The automated classification apparatus for the shoulder diseaseaccording to the present example embodiment may receive the 3D medicalimage, extract features from the 3D medical image, downsample thefeatures and automatically classify the 3D medical image in a pluralityof categories as a diagnosis result.

For example, the categories may include “None” which means that apatient's rotator cuff is not ruptured, “Partial”, “Small”, “Medium” and“Large” which mean a degree of the rupture of the patient's rotatorcuff.

The automated classification apparatus for the shoulder diseaseaccording to the present example embodiment is based on 3D convolutionalneural network (CNN). CNN is a deep learning based artificialintelligence algorithm which shows a powerful performance in analyzingimages. CNN is a deep learning based algorithm which maximizes theperformance of artificial intelligence by deeply connecting anartificial neural network (ANN). CNN includes a lot of learnableconvolutional filters for each connection layer so that CNN learns toextract key features of the image from inputted training data. A basicunit of CNN structure is the convolutional filter. By applying a (1*1),(3*3) or (5*5) filter to the 2D image, a meaningful feature may beextracted from the image. In CNN, these filters are filled with initialrandom values to form a convolutional layer, and as learning progresses,the values of the filters may change to extract the meaningful features.In addition, the convolutional layers are stacked deeply so that thefeatures may be extracted in several stages.

As the convolutional layers are stacked deeply, the donwsampling may beoperated by a pooling operation and adjusting a stride value. In thepooling operation, a most significant value is passed to a next layerfrom a feature map. For example, in a max pooling operation, a maximumvalue in the feature map may be selected. For example in an averagepooling operation, an average value in the feature map may be selected.The stride value may be a parameter of how many pixels the covolutionalfilter moves when the convolutional filter slides the image.

Through the structure that deeply connects the convolutional layersincluding these filters, the artificial intelligence may operate a deeplearning such that the image is analyzed by utilizing from fine featuresof a small area of the image to feature of a large area and a desiredresult is acquired by the analyzing the image. It is the biggest featureand advantage of CNN that CNN analyzes images by viewing such a widereceptive field.

FIG. 2 is a block diagram illustrating a 3D Inception-Downsamplingstructure according to an example embodiment of the present inventiveconcept. FIG. 3 is a block diagram illustrating a 3D Inception-Resnetstructure according to an example embodiment of the present inventiveconcept. FIG. 4 is a block diagram illustrating a 3D Inception-Resnetblock structure according to an example embodiment of the presentinventive concept. FIG. 5 is a block diagram illustrating the automatedclassification apparatus for the shoulder disease according to anexample embodiment of the present inventive concept.

Referring to FIGS. 2 to 5, the automated classification apparatus forthe shoulder disease includes the 3D Inception-Resnet block structuresB53 and B54, a 3D global average pooling structure 3D GAP and B56 and afully connected layer FC and B57 disposed after the 3D global averagepooling structure B56.

For example, in the automated classification apparatus for the shoulderdisease, the 3D medical image B51 may sequentially pass through a first3D convolution structure B52, a first 3D convolution structure B52, afirst 3D Inception-Resnet block structure B53, a second 3DInception-Resnet block structure B54, a second 3D convolution structureB55, the 3D global average pooling structure B56 and the fully connectedlayer B57. In the present example embodiment, the 3D medical image B51may be a 64*64*64 input image.

The 3D Inception-Resnet block structure B53 and B54 may include three ofthe 3D Inception-Resnet structures B41, B42 and B43 and one of the 3DInception-Downsampling structure B44 which are connected in series. Thethree of the 3D Inception-Resnet structures B41, B42 and B43 may havethe same structure. Alternatively, the three of the 3D Inception-Resnetstructures B41, B42 and B43 may have different structures from oneanother.

The 3D Inception-Resnet structure (at least one of B41, B42 and B43) mayinclude a first 3D convolution structure B32, a second 3D convolutionstructure B33 and a third 3D convolution structure B34 connected inseries and forming a first path, a fourth 3D convolution structure B34and a fifth 3D convolution structure B36 connected in series and forminga second path, a concatenate structure B37 concatenating an output ofthe third 3D convolution structure B34 and an output of the fifth 3Dconvolution structure B36 and an add structure B38 operating anelement-wise add operation of the input of the 3D

Inception-Resnet structure and an output of the concatenate structureB37.

The first 3D convolution structure B32 and the fourth 3D convolutionstructure B34 are connected to a previous block B31 and receive theinput of the 3D Inception-Resnet structure B41, B42 and B43.

The 3D Inception-Downsampling structure B44 may include a first 3Dconvolution structure B22 and a maximum pooling structure B23 forming afirst path. The maximum pooling structure B23 may select a maximum valuein the output of the first 3D convolution structure B22. The 3DInception-Downsampling structure B44 may further include a second 3Dconvolution structure B24 and an average pooling structure B25 forming asecond path. The average pooling structure B25 may select an averagevalue in the output of the second 3D convolution structure B24. The 3DInception-Downsampling structure B44 may further include a first stride3D convolution structure B26 including a convolution filter having anincreased moving unit and forming a third path. The 3DInception-Downsampling structure B44 may further include a second stride3D convolution structure B27 including a convolution filter having anincreased moving unit, different from the first stride 3D convolutionstructure B26 and forming a fourth path. The 3D Inception-Downsamplingstructure B44 may further include a concatenate B28 concatenating anoutput of the maximum pooling structure B23, an output of the averagepooling structure B25, an output of the first stride 3D convolutionstructure B26 and an output of the second stride 3D convolutionstructure B27.

The first stride 3D convolution structure B26 may be a 3*3*3 3Dconvolution structure. The stride of the first stride 3D convolutionstructure B26 which means the moving unit of the convolution filter maybe two. The second stride 3D convolution structure B27 may be a 1*1*1 3Dconvolution structure. The stride of the second stride 3D convolutionstructure B27 which means the moving unit of the convolution filter maybe two.

The first 3D convolution structure B22, the second 3D convolutionstructure B24, the first stride 3D convolution structure B26 and thesecond stride 3D convolution structure B27 are connected to a previousblock B21 and receive the input of the 3D Inception-Downsamplingstructure B44.

Referring again to FIG. 2, the 3D Inception-Downsampling structure B44extracts the feature of the previous volume B21 and generates thedownsampled output. In the 3D Inception-Downsampling structure B44, thedownsampling is operated using the pooling and the stride. By pooling,the significant one is selected among the features which are the resultof the convolution. The moving unit of the convolution filter isincreased by the stride, the size of the output may be reduced than theoriginal image.

The results of the downsampled by each method may be all set to have thesame size, so the results are concatenated (B37) like stacking thepapers and the concatenated result are transmitted to a next layer. The3D Inception-Downsampling structure B44 generates a lot of outputfeatures having the reduced size than the previous block B21 so that theresult of the 3D Inception-Downsampling structure B44 may be thecontracted information for a larger range.

Referring again to FIG. 3, the 3D Inception-Resnet structure (at leastone of B41, B42 and B43) is implemented using a 3D convolution filter.The 3D Inception-Resnet structure includes a various types of theconvolution filters B32, B33, B34, B35 and B36 extracting meaningfulinformation from the image received from the previous block B31 or thefeature map. In the 3D Inception-Resnet structure, the size of theoutput passing through each of the (3*3*3) filters may be same as thesize of the input. In the concatenate structure B37, two differentstructures are concatenated so that the features having more variousforms may be extracted.

Referring again to FIG. 4, the 3D Inception-Resnet block structure B53and B54 includes the three 3D Inception-Resnet structures B41, B42 andB43 at a front and the single 3D Inception-Downsampling structure B44 atthe last. Via the structures of FIG. 3 and the structure of FIG. 2, theoutput may be generated by contracting (downsampling) the input image orthe input feature map.

Referring again to FIG. 5, an entire network structure of the automatedclassification apparatus for the rotator cuff tear may include two 3DInception-Resnet block structures B53 and B54. Each of the 3DInception-Resnet block structures B53 and B54 may have the structure ofFIG. 4. The 3D Inception-Resnet block structures B53 and B54 learnfilters which may extract a lot of 3D features which are analyzed in 3D.When the input image B51 passes through the (3*3*3) convolution layersB52 and B55 or the 3D Inception-Resnet block structures B53 and B54, theinformation of the input image B51 may be contracted and the meaningfulfeatures may be extracted so that the final decision may be determined.The single convolution filters B52 and B55 and the 3D Inception-Resnetblock structures B53 and B54 basically have a common characteristic thatextracts features from the image, but the 3D Inception-Resnet blockstructures B53 and B54 may obtain more information than the singleconvolution filters B52 and B55.

Most CNN applied studies are based on 2D images, and practically, a lotof input data are 2D images. However, the medical image such as CT orMRI is a 3D volume image that has image information inside the patient'sbody. A lot of medical image analysis studies using CNN-based algorithmare also actively performed, but it is not possible to fully use therich information of the 3D image because of using the method ofanalyzing multiple 2D images.

In the present example embodiment, the reading of MRI images is trainedusing the 3D Inception-Resnet structure capable of extracting the 3Dfeatures from the image by extending the above-mentioned convolutionfilter of CNN in a three dimension.

The conventional CNN method may have a structure of simply layering theconvolution layers. In contrast, the present Inception-Resnet structuremay combine the structure of Inception structure and Resnet structure tothe convolution layers. In the Inception structure, outputs of thedifferent convolution filters disposed in parallel are concatenated. TheInception structure may represent better results in terms of both acalculation quantity and a performance compared to stacking the samenumber of filters in the conventional method. In the Resnet structure,the output of passing through the several convolution filters and theimage of the previous stage are element-wise added by a residual blockso that the performance of the CNN may be enhanced by keeping theinformation close to the original image in the previous stage.

In the proposed 3D Inception-Resnet structure, the convolution filter,which is the basic unit, is extended to 3D to extract features from the3D volume. The proposed 3D Inception-Resnet structure includes the(1*1*1) filter and the (3*3*3) filter and downsamples the feature map bypooling and stride adjustment. The proposed 3D Inception-Resnetstructure may include the 3D Inception-Resnet structure B41, B42 and B43and the 3D Inception-Downsample structure B44. The 3D Inception-Resnetblock structure B53 and B54 may be generated by combining the 3DInception-Resnet structure B41, B42 and B43 and the 3DInception-Downsample structure B44. The entire network structure of theautomated classification apparatus for the rotator cuff tear may begenerated using two 3D Inception-Resnet structure B41, B42 and B43 andthe 3D Inception-Downsample structure B44.

In order to calculate the 3D Class Activation Map (CAM), which will bedescribed later, the global average pooling (GAP) layer B56 and a singlefully-connected (FC) layer B57 may be disposed at the last stage. TheGAP layer B56 calculates an average of each of the feature maps of theoutput of the last convolution layer. By the GAP layer B56, a weight ineach position may be estimated. The FC layer B57 learns parameters for afinal classification using the output of the GAP layer B56. Although theperformance may be enhanced when the plural FC layers B57 are used, alocation information may be lost while passing the plural FC layers B57.Thus, the single FC layer B57 is used in the present example embodimentfor the CAM calculation. When the number of the FC layer B57 is little,the amount of computation may be reduced so that it may be efficient inthe amount of computation.

According to the present example embodiment, by applying the aboveexplained methods, the performance of CNN shown in the 2D image may beextended to the 3D image. Since the present example embodiment mayefficiently analyze the 3D image of the patient having the largereceptive field, the efficiency of time and cost may be enhanced ratherthan actually making a diagnosis in the medical field as well as ratherthan the conventional methods.

FIG. 6 is a diagram illustrating an operation of a region of interestvisualization part of the automated classification apparatus for theshoulder disease according to an example embodiment of the presentinventive concept.

Referring to FIGS. 1 to 6, the automated classification apparatus forthe shoulder disease may further include the region of interestvisualization part generating a heat map visualizing the region ofinterest identified in the 3D medical image in artificial intelligencegenerating diagnostic results of the 3D medical image.

The region of interest visualization part may generate the heat map bymultiplying the features c1, c2, c3, c4, . . . which are the output ofthe second 3D convolution structure B55 and the weights w1, w2, w3, w4,. . . learned at the fully connected layer B57 and summing themultiplication of the features c1, c2, c3, c4, . . . and the weights w1,w2, w3, w4, . . . . For example, the heat map may be a 3D classactivation map.

FIG. 6 illustrates extraction of the region of interest from an originalimage by the class activation map calculation. The 3D heat map aregenerated by summing the multiplication of the feature maps c1, c2, c3,c4, . . . output from the last convolution and the weights w1, w2, w3,w4, . . . learned at the FC layer.

After the CNN is learned, the feature areas, which the artificialintelligence has seen as significant when making decisions, may bevisualized using the class activation map method.

Since CNN learned to extract many features internally, the visualizationmay be possible using the image for making decisions and the learnedfilter in late layers of the CNN structure. In the case of medicalimaging diagnosis, the visualization of the region of interest isimportant because it is clinically important to explain detaileddiagnosis results beyond simple diagnosis prediction. In the presentexample embodiment, the 3D CNN is used so that the class activation mapmay be calculated in 3D and the 3D visualization may be possible. Byvisualizing the region of interest with MRI data, it is possible to seewhich region is important for predictions made by artificialintelligence.

This visualization not only improves the reliability of the learning andprediction results, but also predicts where the problem occurredclinically.

FIG. 7 is a screen shot illustrating an operation of the automatedclassification apparatus for the shoulder disease according to anexample embodiment of the present inventive concept.

Referring to FIGS. 1 to 7, when the integrated software which iseffective for the learned 3D CNN actual medical field is used, it ispossible to use AI-based diagnostics with proven reliability andstability regardless of time and place.

The software has functions for importing medical data, performing 2D and3D visualization, performing AI-based diagnostics, and visualizing theregion of interest. The importing function reads a Dicom file (having anextension of *.dcm), an image format commonly used in medical images, toreconstruct image and 3D visualization information. When a user only hasa MRI data on the shoulder, the user may check the patient's presence ofrotator cuff tears in real time and may receive the 3D visualizedinformation by simply selecting the largest bone in the shoulder,Humerus, with the mouse, without prior medical knowledge.

FIG. 8 is a table illustrating MRI data of rotator cuff used in anexample embodiment of the present inventive concept. FIG. 9 is a tableillustrating diagnosis result of the automated classification apparatusfor the shoulder disease according to an example embodiment of thepresent inventive concept, diagnosis result of orthopedists specializedin shoulder and diagnosis result of general orthopedists. FIGS. 10 and11 are graphs illustrating diagnosis result of the automatedclassification apparatus for the shoulder disease according to anexample embodiment of the present inventive concept, diagnosis result oforthopedists specialized in shoulder and diagnosis result of generalorthopedists. FIG. 12 is a diagram illustrating how the region ofinterest changes as artificial intelligence learning progresses in theautomated classification apparatus for the shoulder disease according toan example embodiment of the present inventive concept. FIG. 13 is adiagram illustrating rotator cuff tear data and region of interestvisualization of the automated classification apparatus for the shoulderdisease according to an example embodiment of the present inventiveconcept.

Referring to FIGS. 1 to 13, for rotator cuff tear MRI data, aperformance test of the automated classification apparatus for theshoulder disease according to the present example embodiment wasperformed. A total of 2124 MRI images representing presences of tear andsizes of tears are used. Of the 2124 MRI images, 200 randomly sampleddata were designated as test data, and the remaining 1,924 patient datawere used for learning. For the 200 test data, the actual diagnosis wasperformed by 4 orthopedists specialized in shoulder and 13 generalorthopedists to test the accuracy between the automated classificationapparatus for the shoulder disease according to the present exampleembodiment and the orthopedists specialized in shoulder and the generalorthopedists. The artificial intelligence learning classified the imagesinto five categories (None, Partial, Small, Medium, Large) based on thepresence of the rotator cuff tear and the sizes of the rotator cufftears.

As a result of the experiment, the automated classification apparatusfor the shoulder disease according to the present example embodimentrepresents an accuracy of 76.5% in a case of accurately predicting thesize of the rotator cuff tear (Top-1 accuracy). The orthopedistsspecialized in shoulder represents an accuracy of 43.8% and the generalorthopedists represents an accuracy of 30.8% in the Top-1 accuracy sothat the Top-1 accuracy of the automated classification apparatus forthe shoulder disease according to the present example embodiment washigher than the Top-1 accuracy of the orthopedists specialized inshoulder by 32.7% and than the Top-1 accuracy of the generalorthopedists by 45.7%

The automated classification apparatus for the shoulder diseaseaccording to the present example embodiment represents an accuracy of92.5% in a case of predicting only the presence of the rotator cuff tear(Binary accuracy). The orthopedists specialized in shoulder representsan accuracy of 75.8% and the general orthopedists represents an accuracyof 68.3% in the Binary accuracy so that the Binary accuracy of theautomated classification apparatus for the shoulder disease according tothe present example embodiment was higher than the Binary accuracy ofthe orthopedists specialized in shoulder by 16.7% and than the Binaryaccuracy of the general orthopedists by 24.2%

In an aspect of diagnosis time, the automated classification apparatusfor the shoulder disease according to the present example embodimentrepresents high efficiency. It shows that the time required to diagnoseall 200 patient data can be accurately diagnosed in real time with 0.01seconds per person by the automated classification apparatus for theshoulder disease according to the present example embodiment. An averageof 20.7 seconds were required to read one person's data for theorthopedists specialized in shoulder. An average of 31.5 seconds wererequired to read one person's data for the general orthopedists.

As shown in FIG. 12, it is possible to check how the region of interestchanges with 3D visualization data using the 3D CAM method as artificialintelligence learning progresses (as epoch increases).

FIG. 13 represents the data (None, Partial, Small, Medium, Large)including the presence of the rotator cuff tear and the size of therotator cuff tear with the region of interest visualization. By theregion of interest visualization data, the reliability of learning andprediction results may be improved and it is possible to predict whereclinically the problem occurred.

The present inventive concept is related to the automated classificationapparatus for the shoulder disease and the visualization apparatus using3D deep learning, the diagnosis accuracy may be enhanced and thediagnosis time and the diagnosis cost may be reduced.

The foregoing is illustrative of the present inventive concept and isnot to be construed as limiting thereof. Although a few exampleembodiments of the present inventive concept have been described, thoseskilled in the art will readily appreciate that many modifications arepossible in the example embodiments without materially departing fromthe novel teachings and advantages of the present inventive concept.Accordingly, all such modifications are intended to be included withinthe scope of the present inventive concept as defined in the claims. Inthe claims, means-plus-function clauses are intended to cover thestructures described herein as performing the recited function and notonly structural equivalents but also equivalent structures. Therefore,it is to be understood that the foregoing is illustrative of the presentinventive concept and is not to be construed as limited to the specificexample embodiments disclosed, and that modifications to the disclosedexample embodiments, as well as other example embodiments, are intendedto be included within the scope of the appended claims. The presentinventive concept is defined by the following claims, with equivalentsof the claims to be included therein.

What is claimed is:
 1. An automated classification apparatus for ashoulder disease comprising: a 3D (three dimensional) Inception-Resnetblock structure comprising a 3D Inception-Resnet structure configured toreceive 3D medical image of a patient's shoulder and extract featuresfrom the 3D medical image and 3D Inception-Downsampling structureconfigured to downsample information of a feature map including thefeatures; and a global average pooling structure configured to operatean average pooling for an output of the 3D Inception-Resnet blockstructure; and a fully connected layer disposed after the 3D globalaverage pooling structure, wherein the automated classificationapparatus is configured to automatically classify the 3D medical imageinto a plurality of categories.
 2. The automated classificationapparatus of claim 1, wherein the plurality of the categories includes‘None’ which means that patient's rotator cuff tear is not present;‘Partial’, ‘Small’, ‘Medium’ and ‘Large’ according to a size of thepatient's rotator cuff tear.
 3. The automated classification apparatusof claim 1, wherein the 3D medical image sequentially passes through afirst 3D convolution structure, a first 3D Inception-Resnet blockstructure, a second 3D Inception-Resnet block structure, a second 3Dconvolution structure, the global average pooling structure and thefully connected layer.
 4. The automated classification apparatus ofclaim 1, wherein the 3D Inception-Resnet block structure comprises threeof the 3D Inception-Resnet structures and one of the 3DInception-Downsampling structure.
 5. The automated classificationapparatus of claim 1, wherein the 3D Inception-Resnet structurecomprises: a first 3D convolution structure, a second 3D convolutionstructure and a third 3D convolution structure which are connected inseries and forming a first path; a fourth 3D convolution structure and afifth 3D convolution structure which are connected in series and forminga second path; a first concatenate structure configured to concatenatean output of the third 3D convolution structure and an output of thefifth 3D convolution structure; and an add structure configured tooperate an element-wise add operation of an output of the firstconcatenate structure and an input of the 3D Inception-Resnet structure.6. The automated classification apparatus of claim 5, wherein the 3DInception-Downsampling structure comprises: a sixth 3D convolutionstructure and a maximum pooling structure forming a third path, themaximum pooling structure configured to select a maximum value from theoutput of the sixth 3D convolution structure; a seventh 3D convolutionstructure and an average pooling structure forming a fourth path, theaverage pooling structure configured to select an average value from theoutput of the seventh 3D convolution structure; a first stride 3Dconvolution structure including a convolution filter having an increasedmoving unit and forming a fifth path; a second stride 3D convolutionstructure different from the first stride 3D convolution structure,including a convolution filter having an increased moving unit andforming a sixth path; and a second concatenate structure configured toconcatenate an output of the maximum pooling structure, an output of theaverage pooling structure, an output of the first stride 3D convolutionstructure and an output of the second stride 3D convolution structure.7. The automated classification apparatus of claim 1, further comprisinga region of interest visualization part configured to generate a heatmap which visualizes a region of interest identified in the 3D medicalimage in artificial intelligence generating a diagnostic result of the3D medical image.
 8. The automated classification apparatus of claim 7,further comprising a 3D convolution structure disposed between the 3DInception-Resnet block structure and the global pooling averagestructure, wherein the region of interest visualization part isconfigured to generate the heat map by multiplying first features whichare output of the 3D convolution structure and weights learned at thefully connected layer and summing multiplications of the first featuresand the weights.
 9. The automated classification apparatus of claim 8,wherein the heat map is a 3D class activation map.
 10. A method ofproviding information for classification of shoulder disease, the methodcomprising: receiving a 3D (three dimensional) medical image of apatient's shoulder and extracting features from the 3D medical image,using a 3D Inception-Resnet structure; downsampling information of afeature map including the features, using a 3D Inception-Resnet blockstructure; operating an average pooling for an output of the 3DInception-Resnet block structure, using a global average poolingstructure; and automatically classifying the 3D medical image into aplurality of categories.
 11. A non-transitory computer-readable storagemedium having stored thereon program instructions, the programinstructions executable by at least one hardware processor to: receive a3D (three dimensional) medical image of a patient's shoulder and extractfeatures from the 3D medical image, using a 3D Inception-Resnetstructure; downsample information of a feature map including thefeatures, using a 3D Inception-Resnet block structure; operate anaverage pooling for an output of the 3D Inception-Resnet blockstructure, using a global average pooling structure; and automaticallyclassify the 3D medical image into a plurality of categories.